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1.  Fact  Sheet 


1.1  Detailed  Description  of  Human  Infrastructure  Detection  and  Exploitation  (HIDE) 
Army  Technology  Objective  (ATO) 

The  ability  to  use  many  sensors  and  proeess  the  data  at  the  sensor  node  will  provide  new 
intelligenee,  surveillanee,  and  reeonnaissanee  (ISR)  and  Force  Protection  capabilities.  This 
approach  allows  detection  of  targets  which  we  cannot  currently  detect,  improves  ISR  robustness, 
decreases  false  alarms,  and  simplifies  employment  as  compared  to  systems  that  use  single 
modality  sensor  nodes.  This  effort  focuses  on  the  development  of  algorithms,  not  the  sensors, 
applied  to  detection  of  human  infrastructure  presence,  such  as  machinery,  currents  in  wires, 
computer  emanations,  industrial  compounds,  and  humans  themselves  in  confined  spaces,  using  a 
variety  of  low-cost  sensors  including  acoustic,  seismic,  magnetic,  electrostatic  (E-field),  passive 
infrared  (PIR),  chemical,  radio  frequency  (RF),  and  optical  imagers.  The  algorithms  will  be 
structured  to  be  adaptable  to  varying  combinations  of  sensor  modalities,  environmental 
conditions,  and  varying  missions.  The  algorithms  and  sensors  will  be  integrated  on  a  small 
mobile  unmanned  ground  vehicle  (UGV).  The  ATO  will  not  develop  a  platform  or  robotic 
controls,  but  will  leverage  the  ongoing  Command  &  Control  for  Complex  &  Urban  Terrain 
(C2CUT)  ATO  and  other  UGV  efforts,  and  will  integrate  sensors  onto  an  existing  system(s). 
Initial  efforts  will  concentrate  on  a  limited  application  of  detection  of  manmade  machinery  and 
human  activity  in  hidden/confided  spaces.  In  the  latter  part  of  the  ATO,  coordination  with 
Communications-Electronics  Research  Development  and  Engineering  Center  (CERDEC) 
Intelligence  and  Information  Warfare  Director/Directorate  (12 WD)  will  tailor  efforts  for 
transition  to  meet  CERDEC  I2WD’s  Multi-INT  program  requirements.  In  addition,  the  U.S. 
Army  Research  Eaboratory  (ARE)  is  coordinating  this  ATO  with  other  programs,  such  as  the 
Suite  of  Sense-Through- The  Wall  (STTW)  Systems  for  the  Euture  Eorce  ATO  [D]  to  assure 
compatibility  and  no  duplication  of  effort.  In  fiscal  year  (EY)  06,  they  developed  a  test  bed  for 
multi-modal  sensing,  determined  fusion  algorithm  criteria,  and  performed  initial  data  collection 
experiments.  In  EY07,  they  developed  and  evaluated  detection  algorithms;  initiated  fusion 
algorithm  development;  addressed  mobility  and  robotic  platform  issues;  and  performed  field 
experiments.  In  EY08,  they  completed  and  evaluated  the  fusion  algorithms,  and  conducted  a 
final  ATO  demonstration. 

1.2  What  is  the  Problem? 

Einding  human  infrastructure  elements  such  as  machinery,  chemicals,  RE  emissions  or  other 
evidence  of  human  activity  in  confined  enclosed  spaces  is  a  deficiency  in  current  ISR  systems. 

In  addition,  operations  in  these  places  present  a  threat  to  Soldiers  and  detection  of  these  types  of 
presences  is  a  needed  capability. 
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1.3  What  are  the  Technical  Barriers  to  Solving  this  Prohlem? 

•  No  single  sensing  modality  ean  produee  a  robust  ISR  deteetion  eapability. 

•  No  sensor  fusion  algorithms  exist  for  sensor  node  level  0/1  fusion. 

•  There  is  insuffieient  understanding  of  sensor  signatures  and  their  interaetions  in  these 
environments. 

1.4  How  will  you  Overcome  Those  Technical  Barriers? 

•  Collect  hyper-modal  co-registered  sensor  data  and  signatures  in  relevant  environments. 

•  Develop  robust  hyper-modal  sensor  fusion  algorithms. 

•  Develop  a  correlation  matrix  to  establish  relationships  between  sensor  detection 
capabilities  and  relevant  target  signatures. 

1.5  Identify  Alternative  Approaches/Technologies  to  Accomplish/Enhance  ATO 
Objective(s) 

This  approach  will  bridge  the  gap  between  high-cost  large  sensor  systems  and  the  very  low-cost 
systems  currently  under  development.  These  proposed  sensor  systems  will  afford  more 
processing  power  and  a  larger  footprint  due  to  the  fact  that  they  are  mounted  on  a  platform  that 
can  provide  power.  This  ATO  will  not  develop  these  sensors,  only  use  them  to  collect  the  data 
required  to  initiate  sensor  and  fusion  algorithms.  This  effort  will  go  beyond  current  programs  to 
look  at  significantly  more  sensing  modalities  on  a  single  platform.  Development  of  hyper-modal 
low-cost  sensors  on  a  small  mobile  platform  for  ISR  capability  has  not  been  attempted.  ARL  has 
been  discussing  cooperative  interaction  with  Sandia  Laboratories,  Albuquerque,  NM,  on  the 
Virtual  Perimeter  Security  (VPS)  Grand  Challenge  effort  and  integration  of  the  ARL  algorithms 
into  the  Sandia  simulation  models.  ARL  is  also  working  with  CERDEC’s  Night  Vision  and 
Electronic  Sensors  Director/Directorate  to  integrate  into  their  ongoing  modeling  and  simulation 
efforts.  These  models  will  enable  sensor  evaluation  and  location  planning  of  future  sensor 
networks.  ARL  has  also  been  interacting  with  the  United  States  Special  Operations  Command 
(USSOCOM)  to  develop  transition  opportunities.  USSOCOM  is  including  this  effort  in  their 
TTL  Technology  Roadmap.  Applied  research  in  this  ATO  will  leverage  basic  research 
conducted  under  the  Personnel  Detection  Multi-University  Research  Initiative  (MURI),  the 
Microsensors  Collaborative  Technology  Alliance  (CTA),  and  an  internal  ARL  sensor  fusion 
research  effort.  Sensor  fusion  is  a  critical  technology  for  many  future  sensor  systems,  and  ARL 
has  integrated  additional  research  in  fusion  into  the  ARL  research.  We  also  plan  to  leverage 
future  work  done  in  Army  Research  Office  (ARO)  sponsored  MURIs  and  individual  investigator 
projects.  Not  only  is  the  basic  understanding  of  how  humans  detect  and  classify  targets  lacking, 
but  the  fusion  processes  are  even  less  well  understood. 
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1.6  What  is/are  the  Result(s)  of  this  ATO? 

•  An  integrated  hyper-modal  sensor  test  bed  tailored  for  urban  operations 

•  Node-based  algorithms  for  detecting  human  infrastructure  presence  in  hidden/confined 
spaces 

•  A  database  of  co-registered,  hyper-modal  relevant  signatures  and  features  that  are 
detectable  with  available  sensor  technology 

1.7  What  is  the  Potential  Payoff? 

Successful  completion  of  this  technology  will  provide  a  significant  advancement  to  Force 
Protection  and  allow  assessment  of  dangers  in  confined  urban  environments  without  putting 
Soldiers  in  harm’s  way.  In  addition,  successful  completion  of  this  technology  will  enhance 
urban  ISR  capabilities.  The  algorithms  developed  in  this  ATO  will  provide  increased  detection 
and  lower  false  alarm  rates  than  current  capabilities.  Technologies  developed  should  allow 
detection  and  localization  of  threats,  which  cannot  be  done  currently. 


2.  Introduction 


HIDE  is  an  ATO  of  ARL’s  Signal  and  Image  Processing  Division  with  the  objective  of  detecting 
human  infrastructure  and  human  activity  in  difficult  terrains,  such  as  caves,  tunnels,  bunkers, 
etc.,  and  in  urban  environments  using  a  sensor  suite  coupled  with  a  mobile  robotic  platform. 
Recently,  due  to  the  global  war  on  terrorism,  individuals  have  increasingly  become  targets  of 
interest  along  with  their  activity.  So  detection  of  humans  and  human  infrastructure  is  a  main 
focus  for  this  ATO.  In  order  to  obtain  better  ISR,  one  needs  to  obtain  sensor  information  and 
process  it  to  extract  the  right  information  for  ISR.  Traditionally,  video  has  been  the  main  choice 
sensor  for  ISR,  but  it  becomes  useless  if  its  field  of  view  (FOV)  is  limited  and  it  cannot  observe 
occluded  areas.  Similarly,  other  sensors  have  their  limitations,  for  example,  acoustic  sensors  are 
prone  to  poor  performance  if  there  is  loud  background  noise.  Hence,  no  single  sensor  is  capable 
of  solving  the  ISR  problem.  The  best  approach  for  better  ISR  is  to  use  a  multimodal,  low-cost 
multi-sensor  sensing  suite  consisting  of  acoustic,  seismic,  PIR,  chemical,  magnetic,  and  E-field 
sensors.  Since  the  detection  of  the  humans  and  human  infrastructure  is  done  in  remote  locations, 
such  as  caves,  tunnels,  and  urban  areas  that  may  be  hostile,  it  is  appropriate  to  use  a  mobile 
platform  equipped  with  a  multimodal  sensing  suite.  We  used  a  PackBot  that  can  be  maneuvered 
in  difficult  terrains  as  the  platform  for  mounting  sensors.  Some  of  the  sensors  used  are  acoustic, 
seismic,  PIR,  chemical,  magnetic,  and  E-field  sensors  along  with  video  and  infrared  cameras. 
Each  sensor  has  its  own  unique  characteristics  that  complement  with  other  sensing  modalities  in 
obtaining  ISR.  Some  of  the  characteristics  are  listed  below: 
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•  Acoustic  sensors  use  piezoeleetric  microphones  with  a  frequency  response  of  20  Hz  to 
20  KHz.  Acoustic  sensors  are  low-cost  sensors  used  to  detect  any  acoustic  signatures 
generated  by  people  or  machines.  Unlike  some  sensors,  acoustic  sensors  are 
omnidirectional:  they  do  not  require  line  of  sight  to  the  target.  As  a  result,  they  can  be 
deployed  anywhere  in  the  scene  without  worrying  about  line  of  sight.  If  more  than  one 
microphone  is  employed,  acoustic  sensors  can  provide  a  pointing  vector  to  the  direction  of 
the  source  of  the  acoustic  signature. 

•  Seismic  sensors  are  three-axis  sensors  that  can  detect  the  vibrations  in  the  ground.  They 
are  used  to  detect  footfalls,  vibrations  caused  by  machines  being  operated,  etc. 
Accelerometers  can  also  detect  vibrations  in  pipes  produced  by  water  flow.  The  sensitivity 
of  these  sensors  generally  depends  on  the  core  and  the  number  of  turns  used  in  a  coil  that 
moves  in  the  core.  The  frequency  response  also  varies  from  device  to  device.  The  normal 
operating  frequency  range  of  the  seismic  sensor  used  for  this  experiment  is  20  to  100  Hz. 

•  Magnetic  (B-field)  sensors  can  be  used  to  detect  ferromagnetic  materials  carried  by  people, 
e.g.,  keys,  firearms,  and  knives.  These  sensors  may  also  be  used  to  detect  the  usage  of 
computer  monitors. 

•  Electrostatic  (E-field)  sensors  can  be  used  to  detect  the  built-up  electric  charge  on 
personnel.  Together  with  magnetic  sensors,  they  can  also  detect  electrical  activity  in  the 
vicinity,  such  as  the  usage  of  computer  keyboards. 

•  Chemical  sensors  can  be  used  to  detect  the  presence  of  different  kinds  of  chemicals  in  the 
atmosphere,  such  as  pheromones  and  household  chemical  vapors. 

•  PIR  devices  are  very  inexpensive  sensors  that  detect  the  nearby  presence  of  a  warm  body, 
e.g.,  a  human,  within  a  cone-shaped  FOV.  These  sensors  are  basically  motion  detectors. 

•  Visible  imagers  can  capture  color  or  grayscale  video  for  human  gait  detection  and  object 
recognition. 

•  Infrared  imagers  can  detect  and  localize  hot  bodies  and  warm  surfaces,  including  vents  in 
tunnels.  They  can  also  provide  thermal  profiling  of  buildings,  where  warmer  rooms  are 
indicative  of  current  or  recent  human  inhabitation. 
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3.  Sensor  Suite,  Data  Collection  Unit,  and  Mobile  Platform 


In  this  section,  we  present  the  detailed  description  of  the  sensors  and  the  data  collection  unit  used 
in  the  HIDE  ATO.  We  also  describe  some  of  the  features  of  the  mobile  platform  used,  how  the 
sensors  are  mounted  on  the  mobile  platform,  and  the  interface  between  the  sensor  suite  and  the 
platform. 

3,1  The  Sensor  Suite 

Acoustic:  The  acoustic  sensors  used  (figure  1)  are  piezoelectric  microphones.  Acoustic  sensors 
are  low-cost  sensors  used  to  detect  any  acoustic  signatures  generated  by  people  or  machines. 
Unlike  some  of  the  sensors,  these  sensors  are  omnidirectional;  they  do  not  require  line  of  sight  to 
the  target.  As  a  result,  they  can  be  deployed  anywhere  in  the  scene  without  worrying  about  line 
of  sight.  If  more  than  one  microphone  is  employed,  acoustic  sensors  can  provide  a  pointing 
vector  to  the  direction  of  the  source  of  the  acoustic  signature. 


Figure  1.  Single  mierophone  and  an  array  (tetrahedral)  of  microphones. 

Detail  metadata  pertinent  to  this  modality  were  recorded.  The  microphone  used  in  the 
experiment  is  a  Knowles  microphone  that  has  the  frequency  response  20  Hz  to  20  KHz.  The 
data  are  collected  at  16  K  samples  per  second,  which  provides  a  bandwidth  of  8  KHz  that 
captures  both  voice  and  machine  signatures  with  good  fidelity  for  the  HIDE  ATO  purposes. 

Seismic:  Seismic  sensors  are  similar  to  the  acoustic  sensors  and  are  used  to  capture  vibrations 
such  as  (1)  footfalls  when  people  are  walking,  (2)  vibrations  caused  by  the  machines  in 
operation,  and  (3)  water  flow  through  pipes,  to  name  a  few.  Seismic  sensors  measure  the 
vibrations  in  all  three  axes.  The  sensitivity  of  the  sensors  generally  depends  on  the  core  and  the 
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number  of  turns  used  in  a  coil  that  moves  in  the  core.  The  frequency  response  also  varies  from 
device  to  device. 

The  data  are  collected  at  16  K  samples  per  second.  This  is  to  enable  the  sensor  to  capture 
vibrations  from  machines.  Figure  2  shows  the  pictures  of  some  of  the  seismic  sensors.  The  Tri¬ 
axis  seismic  sensor  on  the  right  is  the  one  used  for  data  collection. 


Figure  2.  Seismic  sensors. 

PIR:  PIR  devices  (figure  3)  are  very  inexpensive  sensors  that  detect  the  nearby  presence  of  a 
warm  body,  e.g.,  a  human,  within  a  cone-shaped  FOV.  The  FOV  of  the  sensor  is  determined  by 
the  lens  in  front  of  the  actual  device.  We  employed  two  different  PIRs  one  with  a  15°  FOV  and 
one  with  a  60°  FOV.  We  also  employed  a  sensor  with  charging  plates,  one  charges  positively 
and  another  charges  negatively.  Depending  on  the  direction  of  the  person,  either  the  positive  and 
then  negative  charging  plates  charge  or  vice  versa,  thus  making  it  is  possible  to  determine  the 
direction  of  motion  of  a  target. 

PIRs  are  the  most  reliable  sensors  in  terms  of  detection;  however,  they  do  not  distinguish 
whether  the  target  is  human,  animal,  or  an  inanimate  object.  They  only  record  the  thermal 
signature  generated  by  the  target. 
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Figure  3.  Picture  of  a  PIR  sensor. 


Magnetic  (B-field):  Magnetic  (B-field)  sensors  can  be  used  to  detect  ferromagnetic  materials 
carried  by  people,  e.g.,  keys,  firearms,  and  knives.  These  sensors  may  also  be  used  to  detect  the 
usage  of  computer  monitors.  We  used  two  different  B-field  sensors  (figure  4),  namely, 

(1)  a  fluxgate  magnetometer  and  (2)  a  coil-type  magnetic  sensor.  Three  Quasar  single-axis  8-in 
Model  BS-1004  magnetic  coil  magnetic  sensors  with  a  sensitivity  of  0.1 16  V/nT  (nT  -  nano 
Tesla)  were  used.  They  were  aligned  with  the  x-,  y-,  and  z-axes  to  record  flux  change  in  three 
directions.  The  data  from  these  coils  were  collected  at  256  samples  per  second.  The  three-axis 
fluxgate  magnetometer  used  was  an  Applied  Physics  Systems  Model  1540.  It  has  the  sensitivity 
of  30  pV/nT  and  is  used  to  detect  the  low  frequency  components  up  to  5  Hz.  The  data  from  the 
fluxgate  magnetometer  were  collected  at  10  samples  per  second. 


Figure  4.  Picture  of  the  fluxgate  magnetometer  and  the  coil-type  magnetic  sensor. 
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Electrostatic  (E-field):  The  E-field  sensor  used  on  the  HIDE  ATO  robot  was  an  ES-3000C 
produeed  by  Quasar  Eederal  Systems  (QFS)  (figure  5).  Using  three  pairs  of  sensing  paddles,  this 
sensor  provides  an  analog  output  for  eaeh  of  the  axes,  whieh  is  aeeomplished  by  taking 
difference  measurements  of  the  free-space  potential  at  each  probe  pair.  The  corresponding 
E-field  can  be  determined  by  dividing  by  the  effective  distance  between  the  probes.  Extremely 
low-noise  amplifiers  and  drift  stabilization  circuitry  allow  accurate  measurement  of  the  free- 
space  potential.  The  frequency  response  of  the  ES-3000C  was  from  100  mHz  to  10  kHz,  with  a 
selectable  1-kHz  low-pass  filter.  The  sensitivity  was  10  uV/m/rt  Hz  at  60  Hz  with  selectable 
sensor  gain  settings. 

E-field  data  were  collected  at  8  K  samples  per  second  on  all  three  chaimels,  which  gave  us  the 
fidelity  to  analyze  the  data  carefully. 


Three  Axis  QFS  ES-3000C  E-field  sensor 


Figure  5.  Picture  of  the  three-axis  E-field  sensor. 

Chemical:  Chemical  sensors  are  used  to  detect  the  presence  of  harmful  chemicals  in  the 
vicinity.  The  chemical  sensor  used  in  HIDE  ATO  system  development  was  a  16  channel  multi¬ 
agent  sensor  that  can  detect  16  different  chemicals  (figure  6).  The  sensor  uses  16  different 
wafers  that  react  well  with  the  chemical  they  are  supposed  to  detect.  A  fan  on  the  sensor  diverts 
the  flow  of  air  over  the  wafers.  Depending  on  the  chemical  present,  one  of  the  wafers  reacts  and 
increases  the  resistivity  resulting  in  increase  of  output  voltage. 
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Figure  6.  Picture  of  a  chemical  sensor. 

Visible  and  Infrared  Imaging:  Visible  and  infrared  imaging  sensors  (figure  7)  are  the  most 
widely  used  sensors  in  the  sensing  world,  based  on  the  old  adages  that  a  “picture  is  worth 
thousand  words”  or  “seeing  is  believing.”  The  visible  color  video  camera  had  a  resolution  of 
768  X  494  active  pixels,  Vi-in  Charge-Coupled  Device  (CCD),  and  approximately  450  TV  lines. 
The  FOV  was  approximately  25°.  The  IR  camera  used  was  an  un-cooled  long-wave  (8-12 
micron)  microbolometer  with  a  FOV  of  25°  and  a  resolution  of  160x120  focal  plane  array  (FPA). 
The  data  were  collected  at  2  frames  per  second. 


Figure  7.  Picture  of  the  visible  and  IR  cameras. 
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3,2  Data  Collection 

In  order  collect  the  data  using  all  the  previously  mentioned  sensors,  a  Hyper  Modal  Data 
Collector  (HMDC)  data  collection  unit  was  built  that  was  lightweight  and  could  be  mounted  on  a 
mobile  platform  such  as  a  PackBot.  The  power  for  the  data  collection  unit  came  from  the 
batteries  powering  the  mobile  platform. 

The  following  criteria  were  used  while  designing  the  HMDC: 

•  Scalable — Can  operate  as  a  stand-alone  or  in  arrays. 

•  Expandable — Allows  for  the  addition  of  new  channels/sensors. 

•  Highly  modular — Allows  upgrades  of  individual  sections  without  major  redesign. 

•  Designed  for  field  operation — low  power,  battery  operated,  rugged,  portable,  and  weather 
resistant 

•  Flexible — Designed  to  accept  various  interface  types. 

•  Global  position  system  (GPS)  time  synchronized. 

Figure  8  shows  the  design  details  of  the  HMDC.  Prior  to  building  the  HMDC  for  HIDE  ATO, 
several  deliberations  were  made  with  the  scientists/engineers  involved  on  the  bandwidth 
requirements  for  each  sensor  modality.  It  was  decided,  based  on  the  applications  where  HIDE 
may  be  used,  to  assign  the  sampling  rates  for  each  sensor  modality  as  outlined  in  table  1 . 


HyperModal  Data  Collector 

HIDE  Packbot22  May  2007 


Figure  8.  Details  of  the  FIMDC. 
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Table  1.  Sampling  rate  alloeation  for  each  sensor  modality. 


Sensor  Type 

No.  Channels 

Sampling  Rate 

Resolntion 

(bits) 

Acoustic 

4 -Analog 

16384 

16 

Seismic 

3 -Analog 

16384 

16 

PIR 

1  -  Analog 

256 

16 

B-field  (fluxgate) 

3 -Digital 

10 

24 

B-field  (coil) 

3 -Analog 

256 

16 

E-field 

3 -Analog 

8192 

16 

Chemical 

16-Digital 

1 

Temperature 

1  -Digital 

10 

24 

Each  individual  channel  has  an  anti-aliasing  filter,  and  individual  gain  settings,  which  can  be  set 
using  adjustable  potentiometer.  All  the  data  colleeted  by  various  sensing  modalities  were  time 
synchronized  with  the  GPS.  A  GPS  reeeiver  can  be  plugged  to  the  HMDC  for  GPS  signals.  The 
final  system  is  shown  in  figure  9. 


Figure  9.  FIMDC  unit. 

3,3  Mobile  Platform 

One  of  the  objectives  of  the  HIDE  ATO  is  to  mount  the  disparate  sensors  on  mobile  platform 
that  can  be  easily  maneuvered  in  difficult  terrains  such  as  tunnels,  caves,  and  buildings.  Several 
mobile  platforms,  namely,  mule,  PackBot,  ete.,  were  considered  as  eandidates.  Based  on  the 
availability  and  maneuverability,  we  deeided  to  proceed  with  the  PaekBot  as  the  platform  for 
integrating  the  sensor  suite  for  the  HIDE  ATO.  Eigure  10  shows  the  picture  of  the  PackBot 
before  installing  the  sensors  on  it. 
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Figure  10.  PackBot. 


As  one  can  see,  the  real-estate  on  the  PackBot  is  limited  for  mounting  all  the  sensors  and  the  data 
collection  unit  shown  in  figure  9.  We  decided  to  build  a  platform  on  the  PackBot  to  facilitate  the 
installation  of  all  the  sensors. 


Another  important  issue  that  we  had  to  consider  when  allocating  space  for  each  sensor  was  the 
interference  from  sensors  being  in  close  proximity  to  each  other.  For  example,  the  fan  on  the 
chemical  sensor  can  interfere  with  the  magnetic.  E-field,  acoustic,  and  seismic  sensors.  The 
PackBot  picture  with  all  the  sensors  mounted  is  shown  in  figure  1 1 . 


Figure  11.  PackBot  with  sensors  mounted. 

The  system  in  figure  1 1  is  unique  in  the  sense  that  it  is  the  first  time  so  many  sensors  have  been 
integrated  onto  such  a  small  platform. 
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Data  collected  by  the  HMDC  were  temporarily  stored  on  the  hard  drive  in  the  HMDC  module. 
The  video  and  IR  imaging  sensor  data  were  remotely  collected  on  a  laptop  few  frames  per 
second.  Control  of  the  PackBot  was  done  remotely  using  the  original  video  and  IR  cameras  that 
came  with  the  PackBot  (see  figure  10). 


4.  Data  Collection  for  Phenomenology 


Over  the  span  of  the  HIDE  ATO’s  duration  (3  years),  several  data  collection  exercises  were 
conducted  to  develop  various  fusion  algorithms  based  on  physics-based  phenomenology. 
Algorithms  based  on  phenomenology  are  robust  and  independent  of  application.  Depending  on 
the  type  of  sensor  and  the  target,  several  phenomenological  observations  were  made  by  enacting 
the  right  scenario  during  the  experimentation.  Some  of  the  phenomenology  we  tried  to  observe 
fell  in  two  categories:  direct  observation  and  indirect  observation.  In  the  case  of  direct 
observation,  we  observed  the  targets  face  to  face  while  they  were  in  operation.  In  the  case  of 
indirect  observation,  we  observed  the  targets  after  they  were  powered  down  or  observed  them 
while  in  operation  through  a  wall  or  through  a  screen  to  see  if  these  targets  were  used  recently. 

4,1  Phenomenology  to  Observe 

•  Electromagnetic  radiation  by  use  of  machinery  and  electrical  appliances 

•  Eield  changes  in  the  vicinity  of  the  appliances  and  indoor  and  outdoor  power  feeders 

•  Thermal  profiles  of  the  targets  before  and  after  usage 

•  Acoustic  signatures  of  the  targets  in  use 

•  Seismic  signatures  of  the  targets  in  use 

•  Chemical  signature  of  the  chemical  agents  present  in  the  vicinity 

•  Signature  of  weapons 

•  Signatures  of  common  utility  containers,  utensils,  and  furniture  used  on  a  daily  basis 

Some  of  the  phenomenology  we  tried  to  observe  and  the  sensors  that  may  detect  them  are  listed 
below: 

•  Use  of  machinery  in  a  building,  cave,  etc. :  It  is  possible  to  detect  these  using  acoustic 
(sound),  IR  (hot  surfaces  of  machines  due  to  friction  caused  by  rotating  parts),  seismic 
(vibrations  caused  by  running  machinery;  electrostatic  (rotating  body  and  current  through 
the  wires),  magnetic,  and  video  (detection  of  moving  parts,  etc.). 

•  Water  flow  in  pipes:  When  water  is  being  used  in  a  building,  the  flow  of  water  in  pipes 
makes  distinctive  sounds  and  vibrations.  These  sounds  and  vibrations  may  be  detected  by 
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acoustic,  seismic,  and  accelerometers  when  water  taps  are  opened  and  elosed,  when  a  flush 
is  used,  etc. 

•  Flow  of  air  through  vents:  Temperature  differentials  can  be  deteeted  by  IR,  and  vibrations 
can  be  deteeted  using  seismie  sensors/accelerometers,  and  possibly  by  acoustics. 

•  Chemical  usage:  Ammonia-based  agents  used  for  cleaning,  etc.,  can  be  detected  using 
chemical  sensors.  Other  chemicals,  like  coekroaeh  killer,  methane,  etc.,  might  likewise  be 
detectable. 

•  Footsteps:  Footsteps  can  be  detected  by  seismic  and  acoustic  sensors. 

•  Doors  opening  and  closing:  Doors  opening  and  elosing  can  be  detected  by  seismie, 
acoustie,  IR,  and  video  sensors. 

•  Computer  monitors:  The  use  of  computer  monitors  and  liquid  erystal  displays  (LCDs)  can 
be  picked  up  by  E-field  and  magnetie  (possibly)  sensors. 

•  Telephone  ringing,  radio  playing,  etc.:  These  noises  can  be  deteeted  by  audio  sensors. 

•  Cell  phone  usage:  This  can  be  deteeted  by  RF  and  E-field  sensors. 

We  eolleeted  data  to  develop  various  algorithms  in  the  following  loeations: 

•  Building  507  in  the  Adelphi  Eaboratory  Center  (ALC),  MD 

•  Building  203  in  AEC 

•  Center  for  National  Response 

•  Trailer  in  South  Parking  Eot 

4.1  Data  Collection  in  Building  507  and  Building  203 

This  is  the  first  of  the  several  experimental  data  collection  efforts  undertaken  over  the  lifespan  of 
the  ATO.  The  place  where  the  data  colleetion  will  oecur  was  primarily  selected  due  to  its 
isolation  from  nearby  buildings  and  personnel:  it  is  not  affeeted  by  power  lines,  air-conditioning, 
or  the  day-to-day  aetivities  of  folks  in  an  office  building.  Several  experiments  for  data  eollection 
will  be  conducted  and  their  detailed  descriptions  follow. 

4.1.1  Machine  Shop  Experiment 

The  objective  of  this  experiment  is  to  infer  the  presence  of  people  by  observing  whether  some  of 
the  machines  are  in  use  in  a  loeation  or  building.  In  this  experiment,  we  will  operate  a  drilling 
maehine  and  a  band  saw  (figure  12).  The  objective  is  to  get  the  aeoustie,  seismic,  B-  and  E-field, 
and  thermal  signatures  of  the  maehines  in  use.  After  the  operation  of  the  maehines,  we  will 
determine  how  the  thermal  profiles  of  the  maehines  change  with  time  to  determine  how  long  ago 
these  maehines  were  in  use. 
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Figure  12.  Machine  experiment  setup. 


Experimental  Setup: 

•  Place  IR  imagers  focusing  on  the  drill/band  saw  at  about  16  ft  from  the  instrument. 

•  Place  aeoustie  sensor  within  10-12  ft  from  the  drill. 

•  Place  PIR  sensor  10-12  ft  co-loeated  with  acoustics. 

•  Place  seismie  sensor  10-12  ft  eo-loeated  with  aeoustics. 

•  Place  magnetic  sensors  5  ft  from  the  drill. 

•  Plaee  some  E-field  sensors  near  the  AC  power  cable  feeding  the  drill  press,  some  near  drill 
press,  and  some  near  the  transformer  outside  the  building/underground  power  line. 

•  Place  ehemieal  sensor  5  ft  from  the  drill. 

•  Measure  ground  truth. 

•  Synch  the  elocks  with  GPS  TIME  for  all  sensors. 

•  Collect  data  on  background  noise  (drill  turned  off)  for  2  min. 

•  Turn  on  the  drill  (the  person  should  move  away  from  the  maehine  shop)  and  collect  data 
for  2  min. 
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•  Turn  off  the  drill  (with  the  person  gone)  and  eolleet  data  for  2  min. 

•  Let  the  drill  to  eool  off. 

•  Repeat  the  above  proeess  by  inereasing  the  distanee  by  2x  and  3x,  where  x  is  the  initial 
distance  of  the  sensors  from  the  drill. 

•  Collect  background  noise  after  all  the  above  cycles  are  completed. 

4,1.2  Conference  Room  Experiment 

The  objective  of  this  experiment  is  to  observe  the  people  gathered  around  a  conference  table  and 
the  phenomenology  that  unfolds.  The  phenomenology  may  include  the  thermal  imprints  left  by 
the  people  on  chairs,  tables,  laptops,  etc.  It  also  includes  the  remnant  cigar/cigarette  butts  left 
behind  still  smoldering  or  just  extinguished.  The  experiment  is  used  to  capture  acoustic,  seismic, 
B-  and  E-field,  PIR,  chemical,  video,  and  imaging  sensor  signatures. 

Experimental  Setup: 

•  Place  IR  imagers  focusing  on  the  table/chairs  at  about  12  ft. 

•  Place  acoustic  sensor  within  10-12  ft  from  the  chairs. 

•  Place  PIR  sensor  10-12  ft  co-located  with  acoustics. 

•  Place  seismic  sensor  10-12  ft  co-located  with  acoustics. 

•  Place  magnetic  sensors  5  ft  from  the  chairs. 

•  Place  E-field  sensors  near  the  AC  power  cable,  some  near  chairs,  and  some  near  the 
transformer  outside  the  building/underground  power  line. 

•  Place  chemical  sensor  5  ft  from  the  chairs. 

•  Measure  ground  truth. 

•  Synch  the  clocks  with  GPS  TIME  (possibly  outside). 

•  Collect  data  on  back  ground  noise  (without  any  person  present)  for  2  min. 

•  Three  to  four  people  sit  on  chairs  and  discuss/talk  and  collect  data  for  2  min  while  talking. 

•  Use  a  laptop  for  2  min  while  collecting  data. 

•  All  people  should  leave  the  area.  Collect  data  for  2  min. 

•  Repeat  the  above  process  by  increasing  the  distance  by  2x  and  3x,  where  x  is  the  initial 
distance  of  the  sensors  from  the  table/chairs. 

•  Collect  background  noise  after  all  the  above  cycles  are  completed 
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4.1.3  Bathroom  Scenario 

The  objective  of  this  experiment  is  to  see  if  somebody  is  using  water  in  a  cave  or  building.  If  so, 
running  water  in  pipes  makes  noise  and  causes  vibrations.  We  would  like  to  pick  up  those  and 
more. 

Experimental  Setup: 

•  Place  IR  imagers  focusing  on  the  sink  and  commode. 

•  Place  all  sensors  within  5  ft  from  the  sink. 

•  Measure  ground  truth. 

•  Synch  the  clocks  with  GPS  TIME  (possibly  outside). 

•  Collect  data  on  back  ground  noise  (without  any  person  using  the  bathroom)  for  2  min. 

•  Let  a  person  use  a  sink  or  flush  a  toilet  and  collect  data  for  2  min. 

•  Let  a  person  turn  on  a  light  before  entering  the  bathroom  and  turn  it  off  when  the  person 
leaves.  Data  are  collected  during  this  activity. 

•  Repeat  the  above  process  with  sensors  outside  some  placed  near  pipes  running  along  the 
wall  or  ceiling,  etc. 

4.1.4  Vent  Closing  and  Opening 

The  objective  of  this  experiment  is  to  infer  if  someone  has  opened  a  vent  to  let  fresh  air  flow  into 
a  cave  or  has  turned  on  an  exhaust  fan,  etc.  We  would  like  to  monitor  those  activities,  which  in 
turn  gives  clues  about  human  activity. 

Experimental  Setup 

•  Place  the  magnetic  sensors  directly  below  the  vent. 

•  Place  the  thermal  imager  trained  on  the  vent. 

•  Place  E-field  sensors  at  appropriate  locations  (to  monitor  exhaust  fan  activity,  etc.). 

•  Place  the  acoustic  and  seismic  sensors  close  to  the  vent. 

•  Synchronize  the  clocks  on  test  data  collectors. 

•  Collect  the  background  noise  for  2  min  before  vent  is  opened. 

•  Collect  data  for  2  min  after  vent  is  opened. 

•  Repeat  the  cycle  with  exhaust  fan  turned  on  and  off. 
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4.1.5  Portable  Generator  and  Air  Compressor 


The  objective  of  this  experiment  is  to  see  if  somebody  is  using  a  generator  to  power  the  facilities, 
etc.  This  depends  heavily  on  magnetic  and  e-field  sensors. 

Experimental  Setup: 

•  Set  up  the  magnetic  and  E-field  sensors  as  per  the  expert’s  advice. 

•  Place  acoustic,  seismic,  and  other  sensors  about  10-12  ft  from  the  generator. 

•  Synch  the  clocks  on  the  data  collectors. 

•  Collect  the  background  noise. 

•  Run  a  portable  generator  and  collect  data. 

•  Stop  the  generator  and  collect  data 

•  Do  the  same  with  air  compressor 

4.1.6  Use  of  Keypad  of  a  Computer  and  Use  of  Computer  Monitors 

The  objective  is  to  detect  the  E-field,  magnetic,  and  acoustic  signatures  of  a  keypad  when  it  is 
being  used. 

Experimental  Setup: 

•  Set  up  the  magnetic  and  E-field  sensors  as  per  the  experts’  advice. 

•  Place  acoustic,  seismic,  and  other  sensors  about  5-6  ft  from  the  keypad. 

•  Take  the  ground  truth. 

•  Synch  the  clocks  on  the  data  collectors. 

•  Collect  the  background  noise. 

•  Collect  data  while  typing  on  the  keypad. 

•  Collect  the  data  when  the  monitor  is  off  and  when  it  is  on. 

4.1.7  Personnel  Detection 

The  objective  is  to  detect  people  directly  using  all  types  of  sensors. 

Experimental  Setup: 

•  Set  up  the  sensors  all  around  the  facility. 

•  Take  the  ground  truth. 

•  Synch  the  clocks  on  the  data  collectors. 
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•  Have  people  walk  up  and  down  and  take  data. 

•  Do  aetivities  like  smoking,  typing,  ete.,  for  indireet  deteetion. 

4.1.8  Telephone  Scenario 

The  objective  is  to  detect  usage  of  a  cell  phone/telephone. 

Experimental  Setup: 

•  Set  up  the  sensors  all  around  the  facility;  the  magnetic  and  E-field  sensor  should  be  close  to 
the  telephone. 

•  Take  the  ground  truth. 

•  Synch  the  clocks  on  the  data  collectors. 

•  Collect  background  noise. 

•  Let  somebody  ring  the  phone. 

•  Collect  data  while  the  telephone  is  ringing. 

4.2  Data  Collection  at  Center  for  National  Response 

Center  for  National  Response  (CNR)  is  a  premier  sight  in  the  foothills  of  West  Virginia’s 
mountains  that  has  several  unique  facilities:  (1)  a  simulated  cave,  (2)  a  tunnel,  (3)  a  bunker,  and 
(4)  trailers  for  performing  various  experiments.  This  facility  allows  us  to  simulate  the  facilities 
that  our  Soldiers  may  encounter  in  theater.  A  simulated  cave  may  be  used  for  enacting  the 
scenario  encountered  in  Afghanistan;  a  bunker  may  be  used  to  simulate  the  conditions  fugitives 
may  be  using  in  various  parts  of  the  world  to  hide,  etc. 

The  data  were  collected  over  a  period  of  three  days.  The  first  day  the  data  were  collected  in  a 
bunker.  The  bunker  is  a  cinderblock  room  with  a  concrete  ceiling  mostly  (3/4)  buried  in  the 
ground,  125  in  wide  by  125  in  long  by  83  in  high.  From  the  ground  to  the  center  of  the  front 
microphone  is  97  in.  The  door  in  the  roof  of  the  bunker  is  in  the  south  corner.  There  are  access 
points  for  the  bunker  in  the  southwest  and  southeast  walls. 

For  all  runs  on  Day  1,  the  PackBot  looked  into  the  bunker.  The  PackBot  was  tilted  at  an  angle  of 
-84.4°,  pointing  approximately  northwest.  The  covers  were  open  on  the  top  of  the  bunker  and 
on  the  access  pit  next  to  the  bunker.  Outside  temperatures  were  measured  using  the  dashboard 
thermometer  in  my  Pontiac  Vibe.  Bunker  temperatures  were  measured  using  a  thermometer 
found  in  the  bunker.  The  day  started  out  chilly  and  in  the  shadow  of  the  mountains,  but  by  Run  5 
we  were  in  full  sun.  The  experimental  runs  are  listed  as  following: 

Day  1  Data  Collection  List: 

•  Bunker  Ambient  Data  Collection 
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•  Bunker  with  2  People  Walking  with  Guns,  Talking 

•  Bunker  with  2  Empty  Chairs  Ambient  Data  Collection  (figure  13) 


Figure  13.  Chairs’  thermal  profile. 


•  Bunker  with  2  People  Sitting  with  Guns,  Talking,  Leaving/Cool  Down 

•  Bunker  with  2  People  Sitting,  Using  Laptops 

•  Bunker  with  Coffee  Pot  (figure  14) 


Figure  14.  Burner  before  use  and  in  use. 

•  Bunker  with  Person  Sitting,  Drinking  Coffee,  Smoking  Cigar 

•  Bunker  with  Generator 

•  Bunker  with  Drill  Press  (Outside  Generator)  (figure  15) 


Figure  15.  Drill  press  signatures. 
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•  Bunker  with  Drill  Press  (Battery  and  Inverter  in  Bunker) 

•  Bunker  E-  and  B-field  (No  Imager) 

•  Bunker  with  Chemieals  (Methyl  Salicylate) 

•  Manmade  Objects  in  Natural  Background  (Imager  Only) 


Day  2  Data  Collection  List: 

Day  2  data  collection  was  done  primarily  in  a  simulated  cave.  The  cave  was  actually  a  series  of 
rooms  within  the  tunnel  and  has  a  base  structure  of  2  by  4’s  and  plywood.  The  walls  and  ceiling 
were  covered  in  foam.  Chicken  wire  was  attached  to  the  top  of  the  walls  and  across  the  ceiling 
to  hold  the  foam  in  place.  The  main  chamber  of  the  cave  was  1 1  ft,  9  in  wide,  and  21  ft  long. 

For  reference,  the  entrance  to  the  main  chamber  was  on  the  near  wall,  and  had  a  2  ft,  6  in  wide 
opening,  which  was  1  ft  from  the  left  wall.  There  was  also  an  exit  opening  on  the  far  wall,  which 
was  2  ft,  5  in  wide  and  1  ft  from  the  left  wall.  The  near  and  far  walls  were  1 1  ft,  9  in  and  the  left 
and  right  walls  were  21  ft. 


The  tunnel  was  illuminated  by  two-bulb  fluorescent  lights. 

•  Cave  Entrance  Ambient  Data  Collection 

•  Cave  Main  Chamber  Ambient  Data  Collection 

•  Cave  Sensor  Operation  Test 

•  Cave  Conference  Table  scenario  (figure  16) 


Figure  16.  Conference  room  scene  before  and  after  use. 
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•  Cave  2  People  Sitting,  Using  Laptops 

•  Cave  Radio  Transmissions  Test 

•  Cave  Experiment  with  a  Person  Sitting,  Smoking  Cigar 

•  Imager  Registration  (done  to  ealibrate  the  imaging  sensors) 

•  Cave  Drill  Press  (Outside  Generator)  Operation 

•  Cave  Drill  Press  (Battery  and  Inverter  in  Cave)  Operation 

•  Cave  Experiment  with  Weapons/Movement 

•  Cave  Experiment  with  Spiees  (eumin,  einnamon,  eloves)  on  a  Stove 

•  Cave  Generator  Operation  Experiment 

•  Aeoustie  Calibration  (No  Images) 

Day  3  Data  Collection  List: 

Day  3  began  with  one  run  in  the  Egress  Chamber  to  try  to  deteet  wiring  and  aetivity  through  a 
wall.  We  then  moved  to  the  Training  Support  Eaeility  (TSE)  for  the  remainder  of  the  runs.  We 
ran  our  tests  in  the  Conference  Trailer,  which  contained  a  larger  central  meeting  room  with 
kitchen  facilities,  adjoining  smaller  conference  rooms  on  either  end  of  the  trailer,  and  a  restroom 
on  either  end  of  the  trailer.  There  was  also  a  small  computer  room  set  up  with  five  PCs  with 
network  connections. 

•  Egress  Chamber  -  Human  ActivityA¥iring  through  Wall 

The  conference  trailer  was  used  for  the  remainder  of  the  tests.  In  the  main  room,  the 
background  noises  included  a  refrigerator  running  and  eight  fluorescent  lights  (four  bulbs 
each). 

•  Bathroom  Scenario  Experiment 

The  purpose  of  this  experiment  was  to  detect  any  human  activity  such  as  turning  on  the 
lights  in  the  bath  room,  flushing  of  toilet,  or  running  water  in  the  sink.  We  intended  to 
detect  these  activities  while  they  were  being  carried  out  or  after  they  had  been  carried  out 
by  observing  the  thermal  profile  of  the  various  facilities  such  as  sink,  bath  tub,  toilet  seat, 
etc. 

•  Toilet  Scenario 

The  objective  here  was  to  use  a  toilet  seat  and  determine  its  thermal  profile  to  see  if  it  had 
been  used  recently. 
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Shower  Scenario 


The  objective  here  was  to  use  a  shower  in  the  tub  and  leave.  The  thermal  profile  of  the 
walls,  tub,  and  handles  should  reveal  the  usage  of  the  shower  facility  (figure  17). 


Figure  17.  Shower  seenario. 

•  Use  of  Cleaning  Products 

If  any  of  the  chemical  products  had  been  used  for  cleaning  purposes,  their  presence  should 
be  detectable  in  the  bathroom  for  some  time  after  their  use,  revealing  recent  human 
activity. 

•  Office  (Computer  Room)  Scenario 

In  this  experiment,  data  were  collected  while  normal  activity  was  taking  place  in  an  office, 
such  as  going  into  the  room,  turning  on  lights,  starting  the  computer,  working  on  the 
computer  (keyboard  activity),  etc.  Some  of  the  human  activity  detection  algorithms  were 
developed  to  determine  the  usage  of  these  instruments. 

Notice  in  figure  18  that  the  thermal  image  of  a  person  sitting  in  front  of  the  monitor  is 
reflected  off  of  the  monitor  and  captured  by  the  IR  imager. 
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Figure  18.  Computer  scenario  signatures. 

•  Use  of  Cell  Phone 

Cell  phones  are  ubiquitous.  Their  use  has  gone  up  tremendously  in  reeent  years  and  they 
are  being  used  for  various  purposes.  It  is  highly  useful  to  detect  their  usage  in  a  confined 
area  or  in  an  urban  area.  Detection  of  usage  of  the  cell  phones  obviously  reveals  the 
presence  of  a  person  and  his/her  activity. 

•  Boiling  Water  on  a  Kettle 

The  objective  of  this  experiment  was  to  see  if  there  were  any  kettles  and  to  determine  if 
they  had  been  used  by  somebody  recently. 

•  Use  of  Toaster  and  Microwave  Oven 

The  objective  of  this  experiment  was  to  see  if  there  were  any  toasters  or  microwave  ovens 
and  to  determine  if  they  had  been  used  by  somebody  recently. 

•  Heat  Floor  Vent  and  Heat/ A/C  Intake 

The  objective  of  this  experiment  was  to  determine  if  the  vents  in  a  building  or  a  facility 
were  blowing  either  cold  or  hot  air.  The  thermal  profiles  of  these  could  provide  a  clue  as  to 
the  activity  in  or  occupancy  of  the  facility. 
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4,3  Data  Collection  in  South  Parking  Lot  Trailer 

The  data  collection  in  the  south  parking  lot  trailer  was  primarily  done  for  fine-tuning  the 
algorithms  for  final  demo  purposes.  The  demo  was  done  in  the  trailer  enacting  a  office  room 
scenario.  There  will  be  more  on  this  when  we  discuss  the  demo. 


5.  Algorithm  Development 


Primarily  the  algorithm  development  for  the  HIDE  ATO  is  partitioned  into  two  main  categories: 
(1)  non- imaging  sensor  data  processing  and  (2)  imaging  sensor  data  processing.  In  each  one,  the 
algorithm  is  further  partitioned  into  (1)  personnel  activity  detection  and  (2)  machine  activity 
detection,  as  shown  in  figure  19. 
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Figure  19.  System  flow  chart  for  algorithm  development. 


The  non-imaging  sensors  consist  of  acoustic,  seismic,  PIR,  magnetic,  and  E-field  sensors.  We 
now  present  the  algorithms  for  these  sensors.  As  shown  in  figure  19,  each  algorithm  is  looking 
for  human  activity  and  manmade  objects  such  a  machine,  kettle,  etc. 

5,1  Acoustic  Algorithms 

Personnel  Detection 

In  order  to  detect  the  presence  of  people,  we  try  and  analyze  the  audio  signatures.  If  people  are 
talking,  we  capture  their  voice  data  and,  based  on  the  voice  spectrum  and  their  intensity  levels, 
we  determine  whether  there  is  a  person  or  personnel  present.  Another  way  to  determine  the 
presence  of  people  is  to  perform  speech  processing  of  the  audio  signals  received.  However,  most 
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of  the  speech  processing  software  available  cater  to  only  a  few  languages,  which  is  an 
impediment.  For  this  reason,  we  developed  the  personnel  detection  algorithm  based  on  the 
energy  contents  in  the  voice  spectrum  of  audio  signatures.  The  description  of  the  algorithm  is  as 
follows. 

The  algorithm  first  collects  statistics  with  or  without  people  present  in  the  scene.  The  people  are 
free  to  talk,  walk,  or  simply  be  silent.  We  primarily  concentrated  in  the  frequency  band  between 
1-2000  Hz  for  these  statistics.  Four  different  statistics  were  collected,  one  for  each  band,  1-500, 
501-1000,  1001-1500,  and  1501-2000  Hz.  These  statistics  are  estimated  as  follows:  let 
=  {fi  ,f 2,- represent  the  amplitudes  of  the  spectra  of  the  acoustic  signal  at  instant  t, 
then  the  mean  of  the  spectral  band  is  estimated  as 

<7+500 

Vge  {0,500,1000,1500} 

n=g+l 

Mq  Vg  e  {0,500,1000,1500},  ye  {0,l} 

=std.dev{si^g,S2^q,"-,s,^^\  Vg  e  {0, 500, 1000,1 500},  7  e  {0,l} 

where  E  denotes  the  expected  value  and  std.  dev.  denotes  standard  deviation  and 7=0  corresponds 
to  the  case  with  no  people  present  and 7=1  corresponds  to  the  case  with  people  present.  We  have 
four  means,  corresponding  to  four  bands  and  standard  deviations 

500, 1*500}  for  the  case  when  no  people  were  present  in  the  scene.  Similarly,  we 
have  the  means  and  the  standard  deviations  for  the  case  when  people  are  present.  In  general, 
and  are  the  representatives  of  energy  level  in  each  band  and  their  variances.  Let  us  denote 
the  energy  levels  in  N bands  as  X  ^{Xy, X 2,- where  X,  is  the  energy  in  band  i,  and  we 
assume  the  energy  levels  in  each  band  are  statistically  independent  and  have  the  Gaussian 
distribution  given  by 

^  = - lii  ,1/2  K (2) 

(2^)  \^i\ 

where  My  and  Xi  denote  the  mean  and  variance,  respectively,  and  T  denotes  the  transpose.  Then 
the  likelihood  that  a  person  is  present  or  not  is  given  by 

N 

p{x\H^)=W  p{Xy  I  Hj )  p(Hj  )  7  =  {0, 1}  (3) 

/=1 

where  Hq  and  Hy  are  the  hypotheses  correspond  to  a  person  is  not  present  and  a  person  is 
present,  respectively.  Then  the  posterior  probability  of  human  presence  is  given  by 


26 


p{H,\X) 


(4) 


N 

YIp{X,\H,)  p(H,) 

_ /=1 _ 

N  N 

YIp{X^  \Ho)  p{H,)+Y\p{Xi  \H,)  p{H,) 

/=1  /=1 

Assuming  p{Hq  )  =  p{H^ )  =  0.5 ,  we  can  compute  the  posterior  probability  of  a  human  present  given 
X.  If  it  exceeds  a  particular  threshold  value,  we  declare  that  a  human  is  detected.  The  algorithm 
is  given  below  and  detailed  in  figure  20. 


Algorithm: 

•  Let  ^(0  corresponds  to  1-s  data. 

•  S  =  fft{s)-,  is  the  fast  Fourier  transform  (FFT)  of  the  signal  s{t) . 

•  Compute  the  mean  and  variances  for  each  band  using  equation  1 . 

•  Use  equations  2  and  4  with  appropriate  means  and  variances  for  noise  and  statistics 
collected  on  people  to  compute  the  posterior  probability,  p{Hy  \x). 


•  Use  the  posterior  probability  for  fusion  and  declare  that  a  person  is  detected  if 
p{H^  |X)>0.6. 


Stats,  no  person 


Stats,  with  person 


Figure  20.  Block  diagram  for  the  acoustic  detection  algorithm. 

Programming  Aspect 

To  compute  the  posterior  probability  p{H^  |  x),  we  used  the  “classify”  function  in  MATLAB. 

The  function  takes  three  inputs:  (1)  the  vector  that  is  to  be  classified,  (2)  the  set  of  vectors 
corresponding  to  the  cases  with  and  without  personnel  present,  and  (3)  the  class  of  each  vector  in 
the  second  input.  The  function  “classify”  also  gives  the  percentage  of  wrong  classification — this 
information  is  used  as  the  uncertainty  parameter  for  fusion  using  Dempster-Shafer  fusion 
algorithm,  which  we  describe  later.  One  can  use  several  bands  and  their  energy  levels  along  with 
their  statistics  in  the  above  algorithm  to  improve  the  classification.  The  algorithm  is  modular 
and  can  be  expanded.  We  now  consider  the  acoustic  algorithm  to  determine  the  probability  of  a 
machine  or  machine  activity. 
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Acoustic  Algorithm  for  Manmade  Object  (Machine)  Detection 

The  approach  we  took  for  detection  of  manmade  objects  using  acoustic  sensors  was  to  determine 
the  periodic  signals  in  the  acoustic  data.  The  philosophy  behind  this  is  that  most  manmade 
objects  perform  repetitive  operations  resulting  in  fundamental  frequency  and  its  harmonics.  So 
clearly,  the  algorithm  looks  for  50  Hz,  60  Hz,  and  their  harmonics.  In  fact,  it  looks  for  any  set  of 
harmonics  using  harmonic  line  analysis.  If  two  or  more  harmonics  are  present  in  acoustic  data, 
then  it  is  determined  that  a  manmade  object  is  present. 

Cell  Phone  Ringing  Detection 

Here  we  performed  the  discrete  cosine  analysis  of  the  acoustic  signal  and  determined  the 
frequencies  associated  with  the  ringing  of  cell  phone.  The  presence  of  set  of  frequencies  above 
certain  threshold  associated  with  the  ringing  tone  determines  cell  phone  detection.  The 
algorithm  was  trained  to  determine  these  frequencies  for  several  cell  phone  rings. 

5.2  Seismic  Detection  Algorithms 

Seismic  Algorithm  for  Personnel  Detection 

The  seismic  sensor  is  used  here  to  detect  the  footprints  of  a  person  walking  by  capturing  the 
vibrations  in  the  floor.  A  normal  walk  periodicity  (cadence)  of  a  person  or  gait  frequency  is 
around  2  Hz.  Hence,  several  algorithms  try  to  extract  the  gait  frequency  and  its  harmonics  in  the 
seismic  sensor  data.  In  order  to  detect  the  gait  frequency,  we  used  6  s  of  data  (sliding  scale)  and 
took  its  absolute  value  to  make  the  envelope  of  the  signal  more  predominant.  We  then  took  the 
TFT  of  the  absolute  signal  (FFT  of  the  envelope).  If  more  than  one  person  is  present  in  the 
scene,  the  joint  gait  frequency  may  be  different  and  might  generate  multiple  harmonics.  As  a 
result  we  selected  the  amplitudes  of  the  first  15  bins  of  FFT  corresponding  to  1-15  Hz.  These 
15  amplitudes  became  the  feature  vector.  Just  as  in  the  case  of  acoustic  detection  algorithm,  we 
assumed  these  features  have  the  Gaussian  distribution  and  used  equation  3  for  determining  the 
p{Hq  I  x)  and  p{h^  \x)  with  appropriate  mean  and  variance,  which  were  then  used  in  equation  4 

to  determine  the  posterior  probability.  Note  in  the  case  of  seismic  sensor,  the  N  in  equation  4  is 
15.  We  used  the  posterior  probability  for  fusion  and  declared  that  a  person  was  detected  if 
p[HjX)>Q.6  .  Once  again,  we  used  the  “classify”  function  in  MATLAB  to  compute  the 

posterior  probabilities.  Figure  21  details  the  seismic  detection  algorithm. 
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Stats,  without  people 


Stats,  with  people 


Figure  21.  Seismic  detection  algorithm. 


Seismic  Algorithm  for  Manmade  Object  (Machine)  Detection 

As  for  the  machine  detection,  once  again  we  are  trying  to  detect  the  harmonics  in  the  signals  that 
are  closely  related  to  the  fundamental  frequency  of  operation  of  the  machine.  Since  the  acoustic 
and  seismic  signals  are  very  similar,  we  used  the  acoustic  algorithm  for  manmade  object 
detection  for  the  seismic  signals  as  well. 

5.3  PIR  Detection  Algorithm  for  Personnel 

PIR  sensors  are  typically  used  as  motion  detector;  however,  the  output  of  the  sensor  is 
proportional  to  the  heat  radiated  by  the  body  of  a  person  or  an  object.  As  a  result,  when  a  person 
comes  into  the  range  of  a  PIR  sensor,  it  generates  an  output.  The  PIR  sensors  used  in  this 
experiment  were  dual  plate  sensors,  where  one  plate  got  positively  charged  and  another  got 
negatively  charged.  It  was  possible  to  determine  the  direction  of  a  person  walking  based  on  the 
output  of  the  sensor  depending  on  whether  the  output  changed  from  positive  to  negative  or  vice 
versa.  Since  we  were  only  interested  in  detecting  whether  a  person  was  present  or  not,  we  used  a 
simple  threshold  detector.  Since  this  detector  was,  in  general,  a  very  robust  sensor,  the 
probability  of  detection  p  was  set  to  1  if  it  exceeded  a  threshold  of  0.3  V. 

5.4  Magnetic  Sensor  Algorithms 

There  are  two  types  of  magnetic  sensors  used  on  the  PackBot:  the  fluxgate  magnetometer  and  the 
coil-type  magnetometer.  The  fluxgate  magnetometer  is  a  low  frequency  (10  samples  per  second) 
and  the  coil  type  has  a  sampling  rate  of  256  samples  per  second.  Due  to  their  limited  bandwidth, 
we  could  not  find  any  features  in  the  frequency  domain,  except  the  60  Hz  hum  in  the  data  of  coil- 
type  magnetometer.  The  algorithm  was  developed  for  detection  of  magnetic  material  movement 
or  change  in  the  flux  due  to  electrical  activity  in  the  vicinity  of  the  sensor  is  a  threshold  detector. 
For  this,  the  baseline  noise-floor  was  established  by  collecting  the  ambient  data  prior  to  the 
experimentation  for  several  minutes.  A  threshold  of  0.3  V  was  used  above  the  noise  floor  for 
detection  purposes. 
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5,5  E-field  Sensor  Algorithms 


For  the  limited  experiments  we  conducted,  we  could  not  find  the  phenomenology  based  feature 
set  in  the  frequency  domain  for  personnel  and  manmade  object  detection.  Just  as  in  the  case  of 
magnetic  sensor  algorithms,  we  used  a  threshold  detector  for  the  E-field  sensor  as  the  algorithm 
for  detection  of  anomaly  in  the  E-field  due  to  the  presence  of  a  person  or  machine  activity. 

5,6  Video  Motion  Detection  Algorithm  for  Personnel  Detection 

Another  sensor  used  was  a  video  camera  to  detect  the  presence  of  personnel.  This  was  done  by 
detecting  a  moving  body  in  the  video  frames.  The  camera  captured  the  scene  in  its  EOV  frame 
by  frame  every  second.  Eor  motion  detection,  the  current  frame  was  compared  to  the  previous 
frame.  Timestamps  at  the  bottom  of  the  image  were  removed  before  this  comparison.  The 
images  were  then  subtracted  from  each  other  on  a  pixel-by-pixel  basis  to  find  the  difference. 
The  resulting  difference  “image”  was  filtered  for  noise.  Eirst,  any  small  difference  values  were 
removed  (set  to  0).  The  difference  image  then  underwent  median  filtering,  which  removed  the 
small  elements  of  noise/illumination  changes.  Median  filtering  works  by  making  each  pixel 
equal  to  the  median  of  the  pixels  within  a  small  area.  The  small  area  is  defined  by  a  structural 
element,  which  for  this  algorithm  was  a  4  x  4  neighborhood.  The  total  variance  of  the  resulting 
image  is  computed,  and  if  it  exceeds  a  threshold,  motion  is  detected.  Whenever  a  motion  was 
detected  by  the  motion  detection  algorithm,  the  probability  of  detection  of  personnel  was 
assigned  a  value  of  1 ,  the  probability  of  no  detection  and  uncertainty  were  assigned  a  value  of 
zero. 


5,7  Video  Detection  Algorithms  for  Thermal  Profiling 

One  of  the  aims  for  the  HIDE  ATO  was  to  determine  if  an  object  such  as  a  chair  had  been  used 
recently,  if  so  how  long  ago  the  chair  had  been  used.  In  order  to  do  this,  the  algorithm  had  to  go 
through  several  steps. 

•  Is  it  a  manmade  object? 

•  Registration  of  the  video  and  IR  images 

•  Detection  of  thermal  decay  rate 

•  We  will  address  these  issues  one  by  one. 

Manmade  Object  Detection  Using  Video  Frames 

Manmade  objects  often  have  geometric  shapes  such  as  circles,  triangles,  squares,  etc.,  in  their 
shape.  So  one  natural  approach  is  to  detect  these  shapes  in  the  video  frame  and  identify  all  the 
shapes  that  fit  geometrical  shapes  and  based  on  the  number  of  such  shapes  in  the  frame  and  the 
total  occupancy  of  the  shape  in  the  object  of  investigation,  one  can  determine  the  probability  of 
manmade  objects  present  in  the  video  frame. 
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We  felt  that  manmade  object  detection  in  a  video  frame  is  highly  time  consuming  and 
unnecessarily  ties  up  resources  for  development  of  this  algorithm.  There  may  be  commercially 
available  software  for  detection  of  manmade  objects  that  can  be  used. 

Registration  of  Video  and  IR  Images 

This  is  an  essential  step  that  must  be  performed  to  determine  if  a  manmade  object  is  hot  and  has 
been  recently  used.  We  used  a  standard  registration  technique  used  in  image  processing  for  this. 
Figure  22  shows  the  image  of  chairs  captured  by  video  and  IR  cameras  before  and  after 
registration. 


Figure  22.  Registration  of  video  and  IR  images. 


Thermal  Profiling 

For  this,  we  monitor  the  hot  surface  (in  the  above  example,  a  chair)  and  determine  the  intensity 
level  of  the  heat  with  time.  This  was  done  for  various  metallic  as  well  as  plastic  objects.  We 
developed  a  set  of  thermal  profile  curves  for  various  targets,  which  gave  the  time  elapsed  from 
the  time  the  object  was  last  used. 

5,8  Personnel  Detection  using  IR  Imager 

Thermal  image  in  figure  18  shows  a  thermal  image  of  the  person  in  front  of  the  computer  due  to 
reflection.  Most  of  the  thermal  energy  radiated  by  the  person  in  front  of  the  computer  monitor  is 
reflected  by  the  monitor  and  this  reflected  thermal  energy  was  detected  by  the  IR  imager.  The  IR 
imager  algorithm  processed  the  silhouette  reflected  from  the  monitor.  First,  a  Hough  transform 
{12)  was  used  to  determine  the  line  patterns  of  an  object,  and  then  using  elliptical  and  rectangular 
models  to  detect  a  person  (13-15)  in  front  of  the  monitor  provided  the  probability  of  a  person 
being  present  in  the  room.  Figure  23  shows  the  sequence  of  operations  and  the  resulting  images 
from  the  algorithm.  In  order  to  determine  the  extracted  reflected  image  is  a  silhouette  of  a 
person,  we  extracted  the  dimensions  of  head,  neck,  and  torso.  For  a  person  the  ratio  of  head 
width  to  height  is  -1.2,  similarly  the  ratios  between  neck  width  and  head  height  and  the  neck 
width  to  torso  width  are  used  to  determine  the  probability  of  a  person. 
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Figure  23.  (a)  Thermal  image  of  the  monitor  without  a  person  in  front,  (b)  a  person  in 
front  of  the  monitor,  (c)  the  hot  surfaces  extracted  from  the  image  (b),  and 
(d)  a  silhouette  of  a  person  extracted  from  (c). 


6.  PackBot  Platform  Effect  on  Sensor  Data  and  Processing 


As  mentioned  previously,  acoustic,  seismic,  PIR,  magnetic  field,  E-field,  chemical,  video,  and 
infrared  image  sensors  were  installed  on  the  PackBot  (a  small  robot).  There  were  several 
constraints  on  the  platform  for  integrating  these  sensor  modalities  on  a  small  PackBot.  First  and 
foremost  was  the  real  estate  on  the  PackBot  available  for  mounting  the  sensors.  In  order  to 
mount  all  the  sensors,  we  had  to  install  an  additional  platform  on  the  PackBot.  Another  big 
concern  was  the  use  of  power.  In  order  to  save  the  real  estate  on  the  PackBot  for  the  sensors  and 
the  data  collection  unit,  we  decided  to  use  the  existing  batteries  that  power  the  PackBot  to  power 
the  sensors  and  the  data  collection  unit.  This  helped  to  minimize  the  weight  that  would  have 
otherwise  increased  if  we  had  had  to  mount  additional  batteries  to  power  the  sensors  and  the  data 
collection  unit.  The  data  collection  unit  had  a  limited  number  of  channels  and  the  data  rates  were 
selected  based  on  the  sensors. 

Another  big  concern  was  interference  between  sensors.  For  example,  the  chemical  sensor  had  a 
fan  to  suck  in  the  air  that  flows  over  the  sensor  module  that  reacts  with  the  chemicals  present  in 
the  air.  The  fan  noise  appeared  in  most  of  the  sensor  data.  As  the  fan  motor  rotates,  it  generates 
F-  and  B-field  noise,  which  showed  up  in  the  respective  modalities. 
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In  this  section,  data  from  several  modalities  is  analyzed  and  the  noise  from  the  platform  and 
other  sensors  are  identified.  We  present  some  of  the  techniques  used  to  mitigate  the  noise  due  to 
the  platform  and  other  interfering  sensors  prior  to  applying  the  personnel  detection  algorithms. 
We  found  that  the  mobile  platform  we  used  produced  significant  noise  while  in  motion.  This 
noise  sometimes  saturated  the  sensors.  As  a  result,  the  data  were  collected  mainly  while  the 
mobile  platform  was  stationary.  However,  there  were  other  noises  due  to  interfering  sensors, 
power  lines,  and  the  platform  itself.  Analysis  of  the  data  is  shown  for  all  the  sensors  on  the 
mobile  platform  (the  acoustic,  seismic,  PIR,  and  B-  and  E-  field  sensors).  Video  and  IR  imaging 
sensor  data  were  found  to  be  free  of  interference. 


Acoustic  Sensor  Data  Analysis 


Due  to  the  close  proximity  of  acoustic  sensor  to  other  sensors,  the  data  collected  using  the 
acoustic  sensor  contained  several  interferers  that  were  predominant.  For  example,  some  of  the 
noise  sources  were  due  to  the  chemical  sensor’s  fan  noise,  power  line  hum,  and  the  mobile 
platform  itself.  In  order  to  identify  the  frequencies  of  interference,  we  collected  data  using 
acoustic  sensor  for  several  minutes  at  16  K  sampling  rate  and  performed  EFT  on  the  data  and 
computed  the  average  spectrum  of  the  data  shown  in  figure  24.  From  figure  24,  the  fan  noise  is 
very  predominant  followed  by  the  60  and  120  Hz  hum  due  to  the  transformers  on  the  nearby 
fluorescent  lamps  that  operate  at  60  Hz.  The  mobile  platform  (PackBot)  itself  had  some  inherent 
noise  source  at  14  Hz.  These  frequencies  were  filtered  out  using  a  bank  of  notch  filters  and  the 
average  spectrum  after  filtering  the  noise  is  shown  in  figure  24.  Clearly,  the  noises  are  reduced 
significantly  and  the  data  can  be  processed  for  personnel  detection  or  acoustic  event. 


Acoustic  data  —  Average  spectrum 
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Figure  24.  Acoustic  data  spectrum  before  and  after  noise  removal. 
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Figure  25  shows  the  flow  ehart  for  the  acoustie  signal  proeessing.  The  noteh  filter  box  consists 
of  several  notch  filters  at  60,  120,  230,  460,  690,  and  920  Hz. 
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Figure  25.  (a)  Block  diagram  for  acoustic  data  processing  and  (b)  acoustic  signal  and  probability  of  detection. 
Seismic  Sensor  Data  Analysis 

Seismic  data  were  also  collected  at  16  K  sampling  rate.  We  found  that  the  fan  noise  was  also 
present  in  the  seismic  sensor  data.  Figure  26  shows  the  average  spectrum  of  the  seismic  data 
before  and  after  noise  removal. 


Seismic  data  —  Average  spectrum 
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Figure  26.  Seismic  data  spectrum  before  and  after  noise  removal. 
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We  used  the  algorithm  described  in  section  6.2  for  seismic  sensor  data  for  detection  of  people. 
Figure  27  shows  the  results  of  the  algorithm  for  personnel  detection. 


Figure  27.  Seismic  signatures  and  probability  of  detection. 

PIR  Sensor  Data  Analysis 

This  is  a  motion  detector  with  30°  FOV  lens.  The  data  were  collected  at  256  samples  per 
second.  Since  the  bandwidth  was  128  Hz,  the  fan  noise  (~230  Hz),  which  we  saw  in  both 
acoustic  and  seismic  sensors,  was  not  predominant  in  the  PIR  data.  Just  as  mentioned  in  section 
6.3,  we  used  a  threshold  detector  for  detecting  moving  objects.  Figure  28  shows  the  output  of 
the  algorithm. 
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Figure  28.  PIR  signatures  and  probability  of  detection. 


E-field  Sensor  Data  Analysis 

E-field  sensor  data  were  collected  at  8  K  sampling  rate.  The  harmonics  of  fan  noise  were  more 
predominant  in  E-field  data  compared  to  other  modalities.  Eigure  29  shows  the  average 
spectrum  before  and  after  noise  removal.  Notice  there  is  still  some  residual  noise  signals  due  to 
fan  harmonics  are  left  in  the  average  spectrum  of  the  seismic  data.  This  is  due  to  the  fact  that  the 
fan  noise  frequency  was  changing  with  time  as  the  battery  power  is  changing  due  to  operation. 

So  the  notch  filters  designed  at  specific  average  frequencies  could  eliminate  some  but  not  all  the 
noise  due  to  the  fan. 


Figure  29.  E-field  data  spectrum  before  and  after  noise  removal. 
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The  E-field  algorithm  is  a  threshold  detector  and  its  results  for  the  data  are  shown  in  figure  30. 
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Figure  30.  E-field  data  and  detections  by  the  algorithm. 
B-Jield  Sensor  Data  Analysis 


These  data  were  collected  at  256  samples  per  second;  as  a  result  the  230  Hz  noise  due  to  fan 
noise  was  not  predominant  in  the  B-field  data. 

Once  most  of  the  noise  is  removed  from  the  data  using  several  notch  filters,  it  is  processed  for 
personnel  detection. 


The  B-field  algorithm  is  a  threshold  detector  and  its  results  for  the  data  are  shown  in  figure  3 1 . 
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Figure  3 1 .  Magnetic  field  data  and  detection  by  the  algorithm. 
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7.  Sensor  Fusion 


Sensor  fusion  is  supposed  to  lead  to  a  better  situational  awareness.  However  fusion  of 
multimodal  data  is  a  difficult  thing  to  do  as  few  joint  probability  density  functions  for  mixed 
modalities  exist.  Fusion  mostly  depends  on  the  application  at  hand.  The  problem  is  further 
complicated  if  one  has  to  fuse  the  events  that  take  place  over  a  period  of  time  and  over  a  wide 
area  for  situational  awareness.  If  they  are  time  dependent,  the  relevance  of  the  data  observed  at 
different  times  becomes  an  issue.  We  opted  to  do  a  fusion  of  information  (decision  -  probability 
of  detection  of  an  event).  In  a  majority  of  the  cases,  Bayesian  networks  (4,  5)  were  used  for 
fusion.  In  this  report,  we  use  Dempster-Shafer  fusion  (6,  7)  for  fusion  of  multimodal  multi¬ 
sensor  data. 

So  far  we  have  shown  individual  algorithms  for  individual  sensors  such  as  acoustic,  seismic, 

PIR,  B-  and  E-field  sensors,  and  video  and  IR  imagers.  Each  sensor  is  geared  to  detect  whether  a 
person  is  present  in  the  vicinity  or  not  and  also  determines  whether  any  manmade  objects  are 
present.  As  shown  in  figure  19,  the  data  from  non- imaging  sensors  are  fused  in  the  fusion 
engine.  Eusion  can  be  done  either  at  data  level,  feature  level,  or  information  level.  We  opted  to 
perform  the  fusion  at  information  level.  The  information  each  sensor  algorithm  puts  out  is  the 
probability  of  detection  and  all  the  's  from  individual  sensors  need  to  be  fused  to  come  up 

with  an  overall  probability  of  detection  for  personnel  and  similarly  another  pj  for  manmade 
object  detection. 

We  used  Dempster-Shafer  fusion  paradigm  for  fusing  the  information  from  various  sensors. 
Dempster-Shafer’ s  method  involves  probability  masses  given  by  sensors  to  be  distributed  over  a 
set.  In  this  implementation,  the  set  used  is  {A,  B,  AB}.  “A”  represents  the  evidence  for 
detection,  “B”  represents  the  evidence  against  detection,  and  “AB”  represents  the  uncertainty  or 
ignorance  of  the  sensor. 

Generally,  with  this  scenario,  the  lack  of  evidence  for  detection  does  not  signify  that  there  is  no 
person.  Eor  example,  the  acoustic  sensor  may  not  pick  up  anything  if  the  target  is  intentionally 
being  stealthy  (not  talking).  To  determine  the  uncertainty  of  a  sensor,  we  compute  the 
percentage  of  misclassification  of  training  data  by  each  detection  algorithm  and  use  it  as  the 
uncertainty  for  that  sensor.  We  also  adjust  the  probability  of  detection  and  probability  of  no 
detection  appropriately  so  that  the  sum  total  of  probability  of  detection,  probability  of  no 
detection  and  the  uncertainty  equals  1 . 
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Dempster-Shafer  Rule  of  Combination 

Let  and  S2  denote  two  different  sensors  observing  the  presence  of  a  person  in  their  vicinity. 

The  total  probability  mass  committed  to  an  event  Z  defined  by  the  combination  of  evidence 
represented  by  SfiX)  and  52  (T)  is  given  by 

S,fZ)  =  S,{Z)®SfZ)=K  ^5i(X)52(T)  (5) 

xr\Y=z 

where  K,  the  normalization  factor  is: 

^5i(X)52(F)  (6) 

This  is  basically  the  sum  of  elements  from  the  set  of  Sensor  1  who  intersect  with  Sensor  2  to 
make  Z,  divided  by  1-  the  sum  of  elements  from  5;  that  have  no  intersection  with  S2 . 

The  rule  is  used  to  combine  all  three  probabilities  (A,  B,  AB)  of  sensors  and  S2 .  The 
resultant  probabilities  are  combined  with  the  probabilities  of  the  next  sensor. 

We  used  the  acoustic,  seismic,  PIR,  and  video  motion  detection  algorithms  to  detect  the  presence 
of  personnel  and  computed  the  probability  of  detection,  probability  of  no  detection  and 
probability  of  uncertainty  for  each  sensor  modality  for  the  data.  Figure  32  shows  the  probability 
of  detection  for  all  the  four  acoustic  sensors  on  the  PackBot  and  the  result  of  the  fusion  using 
only  acoustic  sensors  data.  From  figure  32,  it  is  clear  that  the  overall  probability  of  detection  has 
improved  with  fusion. 
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In  figure  33  we  show  the  results  of  fusion  of  aeoustie  and  seismie  data.  Seismie  sensor  deteets 
the  footprints  of  a  person.  From  figure  33  we  notiee  around  the  300-s  time  period  the  seismie 
sensor  has  high  probability  of  deteetion  eompared  to  the  aeoustie  sensors.  This  is  the  ease  where 
the  people  were  not  talking  and  henee  had  a  low  probability  of  deteetion.  We  also  notiee  that  the 
overall  probability  of  deteetion  after  fusion  is  improved.  Figure  34  shows  the  deteetions  by 
individual  sensor  modality.  As  expeeted,  the  deteetions  by  the  seismie  sensor  are  minimum 
eompared  to  other  modalities.  Reliable  deteetions  are  obtained  using  PIR  and  video  sensors. 
Aeoustie  sensors  work  best  when  some  people  are  eonversing. 
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Figure  33.  Probability  of  detection  using  acoustic  and  seismic  data. 


Figure  34.  Detection  before  and  after  fusion. 
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8.  Capstone  Demo 


In  a  situation  where  we  are  monitoring  a  building,  we  would  like  to  know  if  there  is  any  activity 
taking  place.  In  particular,  we  placed  a  robot  inside  an  office  room  (in  stealth  mode,  various 
sensors  will  be  placed  and  camouflaged  to  avoid  detection),  as  shown  in  figure  35.  Figure  35 
also  shows  the  robot  with  all  the  sensors.  The  goal  is  to  assess  the  situation  based  on  the 
observations  of  various  sensor  modalities  over  a  period  of  time  in  the  area  covered  by  the  sensor 
range.  We  enacted  the  data  collection  scenario  with  several  features  built-in  to  observe  the 
happenings  inside  the  office  room  and  assess  the  situation. 


Figure  35.  Demo  scenario. 


Demo  Scenario 

•  A  person  walks  into  the  office  room — this  triggers  the  PIR,  B-  and  E-field,  and  seismic 
sensors. 

•  She  occasionally  talks — the  acoustic  sensor  picks  up  the  voice. 

•  She  sits  in  front  of  a  computer. 

•  She  turns  on  the  computer. 

o  B-  and  E-field  sensors  observe  the  power  surge  caused  by  turning  on  the  computer, 
o  Acoustic  sensors  observe  the  characteristic  chime  of  Windows  turning  on. 
o  The  person’s  movements  are  picked  up  by  the  PIR  sensor. 
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o  Visible  video  shows  a  pattern  on  the  eomputer  sereen  showing  aetivity  on  the 
computer. 

o  The  IR  imager  picks  up  the  reflected  thermal  profile  of  the  person  in  front  of  the 
monitor. 

•  She  types  on  the  keyboard — sound  is  detected  by  the  acoustic  sensor. 

•  She  turns  off  the  computer. 

o  Windows  turning  off  sound  is  observed  by  the  acoustic  sensor. 

o  The  power  surge  after  shutdown  is  observed  by  the  B-field  sensor. 

The  above  mentioned  demo  scenario  was  enacted  and  demonstrated  the  capabilities  of  the  HIDE 
ATO  algorithm  suite. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ARL 

ATO 

C2CUT 

CERDEC 

CNR 

E-field 

EET 

EOV 

EY 

GPS 

HIDE 

HMDC 

12  WD 

ISR 

MURI 

PIR 

QES 

RE 

STTW 

TSE 

UGV 

USSOCOM 

VPS 


U.S.  Army  Research  Eaboratory 

Army  Technology  Objective 

Command  &  Control  for  Complex  &  Urban  Terrain 

Communications-Electronics  Research  Development  and  Engineering  Center 

Center  for  National  Response 

electrostatic-field 

fast  Courier  transform 

field  of  view 

fiscal  year 

Global  Position  System 

Human  Infrastructure  Detection  and  Exploitation 
Hyper  Modal  Data  Collector 

Intelligence  and  Information  Warfare  Director/Directorate 

Intelligence,  Surveillance,  and  Reconnaissance 

Multi-University  Research  Initiative 

passive  infrared 

Quasar  Eederal  Systems 

radio  frequency 

Suite  of  Sense-Through-The  Wall 

Training  Support  Eacility 

Unmanned  Ground  Vehicle 

United  States  Special  Operations  Command 

Virtual  Perimeter  Security 
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NO.  OF  NO.  OF 

COPIES  ORGANIZATION  COPIES  ORGANIZATION 


1  ADMNSTR 

ELEC  DEFNS  TECHL  INFO  CTR 
ATTN  DTIC  OCP 

8725  JOHN  J  KINGMAN  RD  STE  0944 
FT  BEL  VOIR  VA  22060-6218 

1  DARPA 

ATTNIXO  S  WELBY 
3701  N  FAIRFAX  DR 
ARLINGTON  VA  22203-1714 

1  CD  OFC  OF  THE  SECY  OF  DEFNS 
ATTN  ODDRE  (R&AT) 

THE  PENTAGON 
WASHINGTON  DC  20301-3080 

I  US  ARMY  INFO  SYS  ENGRG  CMND 

ATTN  AMSEL  IE  TD  A  RIVERA 
FT  HUACHUCA  AZ  85613-5300 

I  COMMANDER 

US  ARMY  RDECOM 
ATTN  AMSRD  AMR 
WC  MCCORKLE 
5400  FOWLER  RD 
REDSTONE  ARSENAL  AL 
35898-5000 

1  US  ARMY  RSRCH  LAB 

ATTNRDRLCIMG  T  LANDFRIED 
BLDG  4600 

ABERDEEN  PROVING  GROUND  MD 
21005-5066 


3  US  ARMY  RSRCH  LAB 
ATTN  IMNE  ALC  HRR 
MAIL  &  RECORDS  MGMT 
ATTN  RDRL  CIM  L  TECHL  LIB 
ATTN  RDRL  CIM  P  TECHL  PUB 
ADELPHI  MD  20783-1197 

1 5  US  ARMY  RSRCH  LAB 

ATTN  RDRL  CII A 
B  O’BRIEN 
R  GREGORY 

ATTN  RDRL  SEE  P  GILLESPIE 
ATTN  RDRL  SEE  I  D  BEEKMAN 
ATTN  RDRL  SES  M  PATTERSON 
ATTN  RDRL  SES  A 
G  CHATTERS 
J  HOUSER 
R  DAMARLA 
ATTN  RDRL  SES  E 
ALEX  CHAN 
G  SAMPLES 
M  THIELKE 
ATTN  RDRL  SES  P 
A  EDELSTEIN 
G  FISCHER 
JFINE 
S  VINCI 

ADELPHI  MD  20783-1197 

2  CERDEC 

ATTN  12  WD 
M  PELLICANO 
R  FORESTA 

FT  MONMOUTH  NJ  07703 


TOTAL:  26  (I  ELEC,  I  CD,  24  HCS) 
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